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Foreword 


In  2008,  the  Office  of  Naval  Research  (ONR)  funded  the  Personnel 
Integration  of  Selection,  Classification,  Evaluations,  and  Surveys  (PISCES)  effort 
to  address  three  deficiencies:  (1)  reliance  on  independent,  single  use  measures  for 
assessing  the  Navy  Total  Force;  (2)  lack  of  measures  to  classify  and  select 
individuals  for  team  assignments;  and  (3)  lack  of  integration  of  tools  to  measure 
and  predict  Navy  Total  Force  performance.  To  address  the  first  deficiency,  a  set 
of  data  collection  alternatives  is  being  developed,  one  of  which  is  survey  by  text 
message. 

The  Lightning  Poll  is  a  text-messaging  based  option  for  data  collection, 
adapted  from  civilian  use  of  text  messaging.  This  report  is  the  second  of  three  to 
outline  the  Lightning  Poll  portion  of  the  Data  Collection  Alternatives  project,  and 
provides  results  of  the  first-ever  Lightning  Poll  conducted  with  Navy  Recruiters. 
The  first  report  (NPRST  TN  09-6)  discussed  the  background  of,  and  the 
mechanics  for,  a  text  message  survey,  and  the  third  report  will  include  results  of 
the  second  Lightning  Poll  as  well  as  suggestions  for  additional  data  collection 
alternatives. 

The  authors  thank  ONR  for  their  generous  sponsorship.  Also  gratefully 
acknowledged  are  the  thousands  of  Navy  recruiters  who  took  a  few  moments  to 
respond  to  both  the  Lightning  Poll  and  the  follow-up  survey  described  in  this 
report.  In  addition,  the  authors  thank  Mr.  John  Noble,  Mrs.  Sheila  Johnson,  Mr. 
Michael  Evans,  and  Ms.  Sherry  Olive,  all  of  Navy  Recruiting  Command,  for  their 
support  and  enthusiasm  for  the  project. 


David  L.  Alderton,  Ph.D. 

Director 
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Executive  Summary 


Purpose 

To  assist  in  their  duties,  Navy  recruiters  are  assigned  cell  phones  and  are 
authorized  1,000  text  messages  per  month.  However,  Navy  Recruiting  Command 
(NRC)  has  limited  knowledge  of  how  useful  those  text  messages  are  to  recruiters 
and  how  to  better  support  recruiters  in  their  use  of  text  messages.  The  Recruiter 
Lightning  Poll  served  two  purposes:  (l)  to  determine  if  text  messaging  is  being 
used  by  recruiters  and,  if  so,  how  useful  it  is  to  them;  and  (2)  to  determine  if 
surveys  can  be  done  through  text  messaging. 

Approach 

Recruiters  with  Navy-funded  cell  phones  were  sent  letters  notifying  them  that 
they  soon  would  be  receiving  two  questions  as  text  messages.  These  letters 
included  a  Privacy  Act  statement  as  well  as  the  phone  number  from  which  the 
questions  would  be  sent.  About  two  weeks  later,  the  initial  question  was  sent  by 
text  message.  Those  who  replied  automatically  received  the  second  question.  A 
follow-up,  paper-based  survey  was  sent  one  week  after  the  initial  text  messages 
were  sent.  The  follow-up  survey  asked  a  few  additional  questions  about  the  use  of 
text  messaging  as  a  recruiter,  as  well  as  questions  about  the  text  messaging 
survey  process. 

Findings 

About  half  of  recruiters  replied  to  the  initial  text  message,  although  useable 
data  for  both  survey  questions  was  received  from  35  percent  of  recruiters.  Thirty- 
four  percent  of  respondents  completed  the  follow-up  survey,  and  results  were 
linked  to  the  Lightning  Poll  results  by  phone  number. 

Results  show  that  only  15  percent  indicate  that  they  do  not  use  text  messaging 
on  their  Navy  cell  phone.  Over  half  of  recruiters  send  text  messages  to  potential 
applicants  at  least  weekly,  while  87  percent  use  text  messaging  to  keep  in  contact 
with  their  DEPers.  Most  who  use  text  messaging  on  their  Navy  cell  phones  find  it 
“very  useful”  for  doing  their  job. 

In  terms  of  conducting  surveys  through  text  messaging,  almost  all  considered 
the  text-based  survey  easy  to  complete,  and  three-fourths  indicated  they  would  be 
likely  to  complete  other  SMS  surveys.  When  compared  to  other  Navy  surveys, 
most  considered  SMS  to  be  more  convenient  yet  as  secure  as  other  surveys. 
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Recommendations 


Based  on  the  results  of  the  survey,  a  number  of  general  recommendations  are 
offered  for  NRC: 

1.  Continue  allowing  text  messaging  on  Navy  phones,  and  consider 
increasing  the  limit  on  the  number  sent  each  month 

2.  Consider  utilizing  cell  phones  better  designed  for  text  messaging  (e.g., 
those  with  full  keyboard) 

3.  Create  a  training  module  and  accompanying  pamphlet  suggesting  ways  to 
use  text  messaging  for  marketing  purposes  (e.g.,  “The  Navy  ROTC  college 
scholarship  web  site  is  open  for  applications.  Go  to  www.nrotc.navy.com”) 

4.  Utilize  text  messaging  from  NRC  for  notification  of  KEY  policy  updates 

5.  Consider  asking  cell  phone  contractor  to  preprogram  NRC  phone  number 
before  delivery  of  the  cell  phone  to  recruiters  so  recruiters  automatically 
know  message  is  from  NRC  HQ 

6.  Consider  using  Lightning  Polls  for  additional  data  collection  efforts,  such 
as  creating  a  panel  (series  of  successive  surveys  sent  to  the  same  group  of 
respondents)  of  recruiters  or  conducting  surveys  of  DEPers 

7.  Provide  feedback  letter  to  respondents  about  results  and  subsequent 
changes  based  upon  survey  results 

In  addition,  several  recommendations  are  offered  for  future  text  message- 
based  surveys: 

1.  Conduct  additional  Lightning  Polls  to  answer  additional  research 
questions  with  comparison  of  results  to  other  survey  methods 

2.  Ensure  adequate  notification  of  an  impending  survey.  Alternatively, 
consider  using  a  panel  format,  where  respondents  are  selected  and 
notified  once  but  are  requested  to  respond  to  successive  surveys  as  needed 

3.  Consider  providing  an  explanation  that  the  system  is  automated,  and 
therefore  unable  to  respond  personally  to  participants’  questions 

4.  Provide  guidelines  for  completion  (using  a  single  letter  “a”  vs  typing 
“agree”  vs  longer  responses  with  clarifications) 

5.  Send  initial  messages  to  participants  in  smaller  sets  (to  avoid  overloading 
system) 

6.  Vary  possible  answer  choices  between  questions  to  avoid  confusion  about 
which  answer  goes  to  which  question  (i.e.,  a/b/c  for  question  1  and  x/y/z 
for  question  2) 

7.  Contact  cell  phone  companies  to  allow  for  caller  ID  of  Lightning  Poll 
number  on  respondent  cell  phones 
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IX 


Surveys  of  Navy  personnel  have  been  conducted  since  at  least  the  1960s  (see 
Wilcove,  2006).  Through  the  years,  the  media  for  survey  administration  has 
changed,  starting  with  paper-and-pencil  (i.e.,  write-in),  moving  on  to  Scantron 
forms,  and  then  to  web-based.  As  cell  phones  become  increasingly  sophisticated 
and  more  people  have  personal  and  sometimes  work  cell  phones,  it  becomes 
possible  to  move  short,  focused  surveys  to  the  cell  phone,  utilizing  text  messages. 
This  annotated  brief  provides  the  results  of  the  first  known  text  message-based 
survey  of  a  military  population,  conducted  of  Navy  recruiters  in  2008.  Additional, 
backup  briefing  slides  are  provided  in  Appendix  A. 
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Overview 


- 

•  What  is  text  messaging? 

•  NRC  Lightning  Poll 


•  Results  of  NRC  Lightning  Poll  and  Follow-up 
NRC-specific  recommendations 


•  Other  Relevant  Lightning  Poll  Results 
-  Are  Lightning  Polls  a  viable  data  collection  method? 
Recommendations  for  future  Lightning  Polls 


The  brief  is  divided  into  four  key  parts,  beginning  with  a  basic  overview  of  text 
messaging  and  continuing  to  a  discussion  of  the  text-message  survey  (known  as  a 
Lightning  Poll)  conducted  for  the  Navy  Recruiting  Command.  Results  are 
presented  that  are  specific  to  the  Navy  Recruiting  Command  as  well  as  results 
related  to  Lightning  Polls  as  a  survey  methodology  and  data  collection 
alternative. 
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The  first  section  provides  a  brief  overview  of  how  text  messaging  works  from 
the  user’s  standpoint  and  includes  rates  of  use  of  text  messaging  in  both  the 
civilian  and  Navy  population. 


3 


Mai 


Text  Messaging 


Text  messaging  (aka  Short  Message  Service,  or  SMS) 
functions  like  e-mail;  messages  are  sent  to  your  cell 
phone  and  stay  on  the  phone  until  you  delete  them 

Generally  arrive  shortly  after  they  are  sent  (dependent 
upon  cell  phone  reception) 

Messages  are  about  140  characters  long  (actual  length 
dependent  upon  cell  provider) 

Text  messaging  service  plans  vary,  from  unlimited 
messages,  to  a  limited  number  free  and  a  cost  per 
message  thereafter,  to  a  cost  for  each  incoming  and 
outgoing  message 


Text  messaging,  also  known  as  Short  Message  Service  (SMS),  functions 
similarly  to  e-mail  in  that  messages  stay  on  a  cell  phone  until  the  user  takes  some 
action  to  delete  them.  They  generally  arrive  soon  after  they  are  sent,  dependent 
upon  the  wireless  reception  at  the  location  of  each  cell  phone.  Messages  usually 
are  140  characters  long,  although  some  providers  allow  for  160  characters  and 
some  automatically  switch  to  the  Multimedia  Messaging  Service  (MMS)  protocol 
for  longer  messages.  The  cost  of  text  messaging  is  dependent  upon  the  service 
plan  that  each  person  has,  with  the  cost  for  those  without  a  plan  currently  at 
about  20  cents  per  message,  incoming  and  outgoing  (Stross,  2008). 
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Background  -  US  population 


•  As  of  December  2007,  84%  of  US 
population  had  cell  phones  and  48.1  billion 
messages  were  sent  monthly  (source: 

CTIA) 

•  Almost  3A  of  18-24  year  olds  who  have  cell 
phones  use  messaging 

Over  half  of  those  25  and  under  send  or 
receive  a  text  message  daily  (source: 
Pew  Research  Center) 

•  Text  messaging  has  a  variety  of  uses 
Contact  friends 
Coordinate  meetings 
Interact  with  television  (e.g.,  American 
Idol) 

Make  purchases  (e.g.,  Pizza  Hut, 
Amazon.com) 


According  to  The  Wireless  Association  (CTIA,  2008),  84  percent  of  the  U.S. 
population  had  cell  phones  and  48.1  billion  messages  were  sent  monthly,  as  of 
December  2007;  the  numbers  for  June  2008  show  the  cell  phone  penetration 
remains  the  same,  but  the  number  of  messages  has  increased  to  75  billion  per 
month  (CTIA,  2009). 

MessageBuzz  (2008)  provides  mobile  statistics  showing  that  almost  three- 
quarters  of  18-24  year  olds  who  have  cell  phones  use  text  messaging.  Results 
from  the  Pew  Research  Center  For  The  People  &  The  Press  (2007)  show  that  over 
half  of  those  25  years  old  and  younger  send  and/or  receive  a  text  message  daily. 

Individuals,  organizations,  and  businesses  are  using  text  messaging  for  a 
variety  of  uses.  Individuals  may  use  it  simply  to  contact  friends.  Organizations 
may  use  it  to  coordinate  events;  spread  the  word  about  political  candidates 
(Graff,  2008);  send  out  notifications  of  school  closures,  Amber  Alerts,  or  other 
emergency  notifications  (Hodgin,  2009);  or  gather  donations  (Kim,  2009). 
Businesses  outside  the  cell  phone  industry  may  use  it  for  voting  or  prize-winning 
such  as  on  American  Idol  and  other  reality  shows,  advertising  (Richtel,  2009), 
providing  coupons  and  other  incentives/discounts  (Bowman,  2007),  or  ordering 
food  (the  Daily  Green,  2008). 


5 


Background  -  Navy  Population 


In  2005,  61%  of  enlisted  and  82%  of 
officers  who  had  text  messaging 
capabilities  used  them  (source:  2005 
MWR  Survey) 

-  In  2006,  43%  of  DEPers  had  sent  text 
messages 

(source:  2006  DEP  survey) 


In  2008,  64%  of  GenY  enlisted  and  59%  of  GenY  officers  had  sent 
or  received  text  messages  in  the  last  24  hours  (source:  2008 
Retention  Quick  Poll) 

Cell  phones  available  to  the  Navy  through  NMCI  and  GSA  contracts 

Beginning  1 J  uly  2007,  NRC  authorized  1,000  text  messages  per 
month  on  Navy-funded  recruiter  cell  phones 


Results  from  a  Navy  survey  done  in  2005  show  that  the  majority  of  both 
enlisted  and  officers  who  had  text  messaging  capabilities  on  their  phone  actually 
used  them  (Uriell  &  Schultz,  2007). 

In  2006,  the  Delayed  Entry  Program  (DEP)  Survey  showed  that  43  percent  of 
those  in  the  DEP  had  sent  text  messages.  Within  the  2008  Navy,  about  60 
percent  of  both  enlisted  and  officers  who  were  part  of  Generation  Y  had  sent  or 
received  a  text  message  in  the  last  24  hours. 

Within  the  Navy,  cell  phones  are  available  through  both  Navy  Marine  Corps 
Internet  (NMCI)  and  General  Services  Administration  (GSA)  contracts.  In 
addition,  recruiters  were  authorized  as  of  1  July  2007  to  use  1,000  text  messages 
per  month  on  their  Navy-funded  cell  phones,  with  that  number  increasing  to 
unlimited  in  late  2008. 
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Use  of  Cell  Phones  by  Those  in  DEP 

- ^ 


As  of  2005,  45%  of  teens  (ages  12-17)  have  cell  phones,  and  33%  have  used  it  for 
text  messaging. 


The  Pew  Internet  and  American  Life  Project  (2005)  found  that  almost  half  of 
teens  had  cell  phones  and  one-third  of  those  are  texting.  In  comparison,  about  7 
in  10  of  those  in  the  DEP  have  cell  phones  and  43  percent  of  those  are  texting 
friends  and  family. 
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Use  of  Text  Messaging  by  Enlisted 
and  Officers 


In  2005,  93%  of  officers  and  83%  of  enlisted  have  a  personal  or  work  cell  phone  (or 
both).  Thirty-five  percent  of  American  population  uses  text  messaging,  ranging  from 
65%  for  18-29  year-olds  to  8%  for  those  65  and  older.  Seventy-six  percent  of 
enlisted  and  64%  of  officers  have  text  messaging  capabilities  on  their  cell  phones. 
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Publicity  Section 


I  Enlisted  ■  Officer 
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Note:  Graphs  show  results  for  those  with  cell  phones  and  text  messaging  capability. 
2005  Navy  MWR  Customer  Survey 


Civilian  findings  (Pew  Research  Center,  2006)  show  that  while  35  percent  of 
the  American  population  uses  text  messaging,  younger  people  are  much  more 
likely  to  use  it  (65%)  than  older  (8%).  The  Navy  population  tends  to  be  younger 
than  the  civilian  population,  so  the  finding  from  the  2005  MWR  Survey  (Uriell  & 
Schultz,  2007)  that  61  percent  of  enlisted  use  text  messaging  is  not  surprising. 

Additionally,  about  one-quarter  of  the  Navy  at  that  time  indicated  that  they 
would  be  interested  in  receiving  messages  from  the  Morale,  Welfare  and 
Recreation  program  and  about  10  percent  would  be  interested  in  participating  in 
a  short  text  message  survey. 
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The  2007-2008  Retention  Quick  Poll  found  that  older  Sailors  (“Boomers”)  are 
less  likely  to  have  used  text  messaging  in  the  last  24  hours  when  compared  to 
both  Generation  X  and  Generation  Y,  similar  to  findings  in  the  civilian 
population. 
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■St 


Text  Message  Authorization  for 
Navy  Recruiters 


Navy  Recruiters  Get  Text  Messaging 

Week  of  July  09,  2007 

The  commander  of  Navy  Recruiting  Command  (NRC)  has  authorized 
Navy  recruiters  the  full  use  of  text  messaging.  Although  text 
messaging  has  never  been  against  the  rules  in  recruiting,  it  has 
always  been  "strongly  discouraged"  due  to  cost  issues.  Recent 
budget  approvals  have  allowed  recruiting  districts  to  authorize 
official  cell  phones  with  text  messaging  plans,  typically  allowing  a 
recruiter  1,000  text  messages  a  month.  However,  the  messages  are 
not  without  normal  restrictions.  Texting  falls  under  normal  "official 
government  business"  use  rules. 


http:/  /  www.militarv.com/  military-  report/  naw-recruiters-aet-text-messaaina?ESRC=miltrep.nl 


In  2007,  Navy  Recruiting  Command  approved  the  use  of  text  messaging  by 
Navy  recruiters  as  part  of  their  normal  government  business.  There  are  a  few 
different  providers  and  many  different  cell  phones  provided  to  recruiters.  The 
majority  were  on  either  Verizon  (56%)  or  Nextel  (38%)  networks,  using  phones 
such  as  those  shown  below. 
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A  Lightning  Poll  is  a  text  message-based  survey  (see  Uriell  &  Clewis  [2009] 
for  details  on  planning,  programming,  and  analyzing  a  text  message-based 
survey).  This  section  details  the  particulars  of  the  Lightning  Poll  conducted  for 
the  Navy  Recruiting  Command. 
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■Sc,  I 


Objective  of  Lightning  Poll 


NRC  research 

Are  text  messages  being  used  by  recruiters? 

Is  text  messaging  useful  to  recruiters  in  doing  their  job? 

Should  NRC  invest  in  the  capability  to  send  "alert" 
messages  to  recruiters  (e.g.,  notifying  them  of  key  policy 
changes)? 

ONR  research,  funded  through  the  PISCES  (Personnel 
Integration  of  Selection,  Classification,  Evaluations  & 
Surveys)  initiative 

-  Can  surveys  (Lightning  Poll)  be  done  through  text 
messaging? 

Do  text  messaging  surveys  obtain  results  comparable  to 
traditional  means;  are  there  differences  for  unique 
groups? 


The  current  Lightning  Poll  had  both  operational  and  research  objectives. 

On  the  operational  side,  the  Navy  Recruiting  Command  was  interested  in 
information  related  to  their  business  processes,  including  metrics  about  use  of 
text  messaging  by  recruiters,  usefulness  of  text  messaging  to  recruiters,  and 
opinions  from  recruiters  on  whether  headquarters  should  be  able  to  send  a  mass 
text  message  to  all  recruiters. 

On  the  research  side,  there  were  two  additional  research  aspects  of  the 
Lightning  Poll  that  were  investigated  for  the  Office  of  Naval  Research  through  the 
Personnel  Integration  of  Selection,  Classification,  Evaluations,  and  Surveys 
(PISCES)  initiative:  can  surveys  be  done  through  this  method  and  are  results 
comparable  to  traditional  assessments  for  similar  groups. 
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Method 


> 

•  Recruiters  with  Navy-funded  cell  phones  were  sent  letters 
notifying  them  that  they  would  soon  be  receiving  two  questions 
on  the  cell  phone  about  their  use  of  text  messaging 

Privacy  Act  statement  accompanied  notification  letter, 
informing  recruiters  that  the  survey  was  voluntary 


-  Letter  included  phone  number  from  which 
messages  would  be  sent 


Initial  text-message  survey  question 
was  sent  by  text  message  about  2 
weeks  later 

Second  message  was  only  sent  if 
respondent  replied  to  first  one 


Recruiters  are  provided  with  cell  phones  by  the  Navy  and  given  an  allowance 
of  text  messages  (1,000  at  the  time  of  the  survey  administration).  A  list  of  all 
recruiter  cell  phone  numbers  was  provided  along  with  respondent  Social  Security 
Number  (SSN),  so  that  current  command  address  could  be  found  for  each 
recruiter.  A  letter  (Appendix  B)  was  sent  to  all  recruiters  at  their  command 
addresses  notifying  them  that  they  would  soon  be  receiving  two  questions  from 
the  Lightning  Poll  cell  number  about  their  use  of  text  messaging.  The  envelope 
also  included  a  Privacy  Act  statement,  notifying  them  of  their  rights  as  well  as  the 
survey’s  Report  Control  Symbol,  which  is  the  Navy  survey  license  for  the  project. 

About  two  weeks  later,  the  initial  text  message  question  was  sent  to  all  cell 
phone  numbers.  The  second  text  message  containing  the  second  and  final  survey 
question  was  only  sent  if  the  respondent  replied  to  the  first  question. 
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Method  (continued) 


I- 

in 

a: 


•  Follow-up  survey  was  sent  a 
week  after  text- message 
survey  began 

Paper-based  survey  asked 
about  text-message 
survey  process  and 
included  additional 
questions  about  text 
messaging  as  a 
recruiter 


About  a  week  after  the  initial  text  messages  were  sent,  respondents  were  sent 
a  paper-based  follow-up  survey  (Appendix  C).  The  paper-based  survey  asked 
opinions  about  the  survey  process  as  well  as  additional  questions  about  the  use  of 
the  text  messaging  by  recruiters.  The  follow-up  survey  was  paper-based  to  help 
ensure  that  all  respondents  would  be  able  to  participate. 
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Lightning  Poll  Technology 

> 

•  Programmed  in  Pangolin's  I  MU  Pro  software  (version  5,  build  1328) 

Similar  to  sending  email  in  Outlook 

-  Used  Trivia  Task  within  software  to  automatically  send  second 
question 

•  First  question  sent  as  broadcast  to  all  recruiter  cell  numbers 

Subsequent  messages  only  sent  if  a  response  received 

Large  number  of  initial  outgoing  messages  may  have  delayed 
delivery  of  initial  messages 

•  Outgoing  and  incoming  messages  exported  from  I  MU  into  SPSS  to 
determine  survey  progress  for  each  respondent 


The  Lightning  Poll  conducted  for  the  Navy  Recruiting  Command  used 
Pangolin’s  IMU  Pro  software,  specifically  version  5,  build  1328.  The  software 
handles  text  messaging  in  a  way  similar  to  sending  an  e-mail  in  Microsoft 
Outlook;  messages  are  sent  to  users  and  stored  within  folders.  Because  the 
Lightning  Polls  involve  questions  in  succession,  the  Trivia  Task  was  used  to 
automatically  handle  the  second  question. 

The  first  question  was  queued  to  all  recruiter  cell  phones,  although  they  were 
dispatched  individually.  Therefore,  there  was  a  long  time  delay  between  the  first 
recruiter  receiving  the  first  question  and  the  last  recruiter  receiving  that  question 
This  also  led  to  a  delay  in  the  second  question  being  sent,  because  the  system  is 
designed  to  process  all  messages  in  the  queue  before  sending  out  additional 
messages. 

Once  the  survey  period  closed,  all  outgoing  and  incoming  messages  were 
exported  from  IMU  and  then  imported  into  SPSS  for  analysis. 
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■St 


Survey  Administration 


Notification  letter  sent: 

Initial  text  message  sent: 
Follow-up  survey  sent: 

Text  message  field  closed: 
Follow-up  survey  field  closed: 


May  29,  2008 
J  une  9,  2008 
J  une  16,  2008 
June  18,  2008* 
October  6,  2008 


Number  sent  and  received 

Eligible  Sample: 

4,620 

Number  Who  Replied  to  Text: 

2,317 

Undeliverable  Surveys: 

273 

Returned  Follow-up  Surveys: 

1,479 

Response  Rates 

Text  Survey: 

49%** 

Follow-up  Survey: 

34% 

*  Field  actually  left  open  until  the  follow-up  survey  field  closed,  but  included  analyses  only  based  on  text 
messages  sent  and  received  prior  to  June  18,  2008. 

**  Based  on  those  who  replied  anything  to  first  question. 


The  initial  notification  letter  was  sent  in  late  May  of  2008,  with  the  initial  text 
messages  being  sent  about  two  weeks  later.  Because  Lightning  Polls  were 
conceived  of  as  a  faster  data  collection  method,  it  was  assumed  that  the 
administration  time  period  would  only  be  open  for  a  week.  The  text  message 
survey  was  actually  allowed  to  remain  open  until  the  follow-up  survey  was  closed, 
but  results  of  this  brief  are  presented  assuming  that  it  was  only  left  open  for 
about  a  week,  or  until  respondents  would  likely  have  received  the  follow-up 
survey;  only  57  recruiters  are  not  included  in  results  presented  here. 

Four  thousand  six  hundred  twenty  phone  numbers  were  sent  the  initial  text 
message.  Of  those,  2,317  provided  some  reply,  yielding  a  return  rate  of  49 
percent.  For  the  follow-up  survey,  1,479  replied,  yielding  a  return  rate  of  34 
percent.  Typical  Navy-wide  survey  response  rates  are  currently  30%  or  less. 
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The  population  of  recruiters  at  the  time  of  the  Lightning  Poll  was 
predominantly  male  First  or  Second  Class  Petty  Officers  (POi  and  PO2, 
respectively).  Results  are  presented  overall  as  well  as  by  gender  and  paygrade 
group. 

A  third  demographic  category  was  created  based  upon  the  results  of  two 
questions  on  the  follow-up  survey.  Recruiters  were  defined  as  a  “heavy-texter”  if 
they  sent  30  or  more  messages  weekly;  about  one-third  of  recruiters  did  so. 

The  table  below  shows  the  percentage  of  recruiters  with  the  indicated  text 
messaging  behavior  on  both  the  Navy-funded  and  their  personal  cell  phones  of 
recruiters. 


Texting  habits  on  personal  cell  phone 
(based  on  average  number  of  text  sent  weekly) 


Don't  use 
text 

Light  user 
(1-29) 

Heavy  texter 
(30+) 

Texting  habits  on 
Navy-funded  cell 

Don't  use 
text 

8% 

4% 

3% 

phone  (based  on 
average  number 

Light  user 
(1-29) 

12% 

23% 

20% 

of  text  sent 
weekly) 

Heavy  texter 
(30+) 

4% 

7% 

20% 
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SMS  Analyses 


Some  text  message  replies  wrapped  into  two  messages  due  to 
character  limit,  but  automated  system  acted  as  if  it  was  two 
separate  replies 

Where  possible,  entire  dialog  with  respondent  was  used  to 
determine  answers  to  the  two  text  message  questions 

Occasionally,  1st  response  was  "Who  is  this,"  so  the  order  of 
the  remainder  of  responses  was  questionable  and  therefore 
not  included  in  analyses 


3606  How  often  do  you  text  one  of  your  potential 

3606  ...  Whosthis? 

3606  Text  messaging  assists  me  in  keeping  in  tou... 

3606  ...  C 

3606  Text  messaging  assists  me  In  keeping  in  tou... 


3579  How  often  do  you  text  one  of  your  potential 

3579 

Who  is  this? 

3579  Text  messagng  assists  me  in  keeping  in  tou.. 

3579 

How  often  do  you  text  your  potential  recr 

3579  Survey  complete  Thank  you  for  your  tone 

3579 

A...  Daly 

If  respondent  included  answer  in  letter  as  well  as  words,  but  the 
two  disagreed  (e.g.,  "a.  Neither"  instead  of  "c.  Neither"),  the  word 
was  used  as  the  answer 


As  this  was  the  first  text  message  survey  conducted,  there  were  a  few 
unforeseen  issues  in  the  responses  that  needed  to  be  resolved  prior  to  analyzing 
the  results.  Some  respondents  sent  back  the  entire  question  with  their  answer, 
leading  to  messages  that  were  more  than  140  characters.  These  long  messages 
were  divided  into  multiple  messages  by  the  cell  phone  provider,  which  confused 
the  automated  survey  system  into  thinking  that  it  had  received  replies  to  the  two 
questions.  The  “conversation”  with  each  respondent  was  read  to  determine  if  two 
actual  answers  were  received. 

Occasionally,  a  respondent  would  reply  to  the  first  question  with  “Who  is 
this,”  perhaps  indicating  that  the  respondent  did  not  receive  the  notification 
letter  or  did  not  remember  what  the  phone  number  was  for  this  survey  effort. 
Because  there  was  no  “correct”  answer  for  each  survey  question,  the  automated 
system  assumed  that  this  was  the  answer  and  sent  the  second  question.  It  was 
then  unclear  if  their  first  letter  answer  was  in  response  to  the  first  question  they 
received,  or  in  response  to  the  question  they  had  just  received.  Coupled  with  this 
was  a  glitch  where  some  respondents  inadvertently  received  multiple  text 
messages  with  the  same  question.  The  slide  above  provides  a  visual  display  of  the 
resulting  data.  On  the  left  side,  it  is  unclear  if  the  “C”  answer  is  the  answer  to  the 
first  question  or  the  second  question,  whereas  on  the  right  side  it  is  clear  that  the 
answer  to  the  first  question  actually  is  “A.  Daily”  since  the  second  question  did 
not  have  “Daily”  as  an  answer  option.  As  indicated  above,  all  text  message 
conversations  were  analyzed,  and  those  where  there  was  doubt  about  the  answer 
were  discussed  by  two  analysts;  if  it  was  unclear  which  letter  answer  was  in 
response  to  which  question,  both  were  marked  as  blank. 
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A  few  respondents  opted  to  include  a  letter  answer  as  well  as  the  written 
answer,  with  the  two  sometimes  not  in  agreement.  In  cases  such  as  this,  the  word 
was  used  as  the  respondent’s  answer  instead  of  the  letter. 
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The  results  of  both  the  Lightning  Poll  and  the  follow-up  are  separated  based 
upon  the  research  interest.  This  section  provides  results  specific  to  and  actionable 
by  the  Navy  Recruiting  Command. 
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How  Often  Recruiters  Send  Text 
Messages  to  Potential  Applicants 


The  majority  of  recruiters  indicate  that  they 
contact  potential  applicants  at  least  monthly 
through  text  message. 


10  13 


Daily 


Weekly 


Monthly 


Rarely 


*  Only  includes  those  with  answers  (1,864)  that  could  be  categorized  into  a 
single  category.  An  additional  165  provided  "Other"  comments,  said  it 
depends,  or  selected  multiple  answers. 

Lightning  Poll,  Q1 


The  first  text  question  is  shown  in  the  cell  phone  window  displayed  above, 
asking  how  often  recruiters  sent  text  messages  to  potential  applicants.  The 
majority  of  the  1,864  responses  that  could  be  categorized  indicated  that  recruiters 
contact  potential  applicants  at  least  monthly  through  text  message.  About  one- 
third  of  recruiters  contact  potential  applicants  on  a  daily  basis  through  text 
messaging. 
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How  Often  Recruiters  Send  Text 
Messages  to  Potential  Applicants 


Little  difference  between  genders 
but  larger  difference  between 
paygroups  and  type  of  texter. 


Gender 


Type  of  Texter 


|  ■  Non-user  in)  Light  user  B  Heavy  personal  □  Heavy  Navy  □  Heavy  user] 

Paygrade  Group 


Daily  Weekly  Monthly  Rarely 


|  B  Male  □  Female  | 

*  Only  includes  those  with  answers  that  could  be  categorized  into  a  single  category. 
Lightning  Poll,  Q1 


Results  for  the  first  text  message  question  were  analyzed  by  gender,  type  of 
texter,  and  paygrade.  There  was  little  difference  in  responses  between  men  and 
women.  However,  there  were  larger  differences  by  type  of  texter;  not  surprisingly, 
those  who  are  heavy  texters  on  their  Navy  phones  are  most  likely  to  indicate  they 
send  messages  daily  to  potential  applicants  while  those  who  do  not  use  text 
messaging  are  most  likely  to  never  send  them  messages.  Additionally,  there  are 
differences  by  paygrade  group;  first  and  second  class  petty  officers  are  more 
likely  than  officers  to  send  text  messages  to  applicants  on  a  daily  or  weekly  basis. 
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Text  Messaging  Assists  Recruiters 
to  Keep  in  Touch  with  DEPers 


Almost  all  agree  that  text  messaging  helps  to 
stay  in  contact  with  DEPers. 


Agree 


Disagree 


*  Only  includes  those  with  answers  (1,  599)  that  could  be  categorized  into  a 
single  category.  An  additional  41  responses  were  "Other." 

Lightning  Poll,  Q2 


The  second  text  message  question  asked  recruiters  if  they  felt  that  text 
messaging  assisted  them  in  keeping  in  touch  with  those  in  the  Delayed  Entry 
Program  (DEPers).  The  large  majority  (87%)  of  the  1,599  categorizable  responses 
indicated  agreement. 
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Text  Messaging  Assists  Recruiters 
to  Keep  in  Touch  with  DEPers 


Heavy  texters  on  Navy  cell  phones 
are  more  likely  to  agree  that  text 
messaging  helps  maintain  contact 
with  DEPers. 


Gender 


Type  of  Texter 


|  ■  Non-user  £3  Light  user  B  Heavy  personal  D  Heavy  Navy  □  Heavy  user  | 

Paygrade  Group 


Male  B  Female  | 


|  □  PQ2  B  POl  □  CPO  □  Off.  | 


*  Only  includes  those  with  answers  that  could  be  categorized  into  a  single  category. 
Lightning  Poll,  Q2 


Again,  there  were  few  differences  in  opinions  between  men  and  women,  but 
large  differences  based  upon  type  of  texter  and  the  paygrade  group.  Only  non¬ 
users  were  noticeably  different  in  their  responses;  they  were  more  likely  to 
indicate  “Neither”  or  “Disagree”  when  compared  to  the  other  four  texting  groups. 
As  seen  for  the  previous  question,  officers  are  less  likely  to  endorse  this  than  the 
First  and  Second  Class  Petty  Officers. 
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Mai 


Use  and  Usefulness  of  Navy- 
funded  Text  Messages 


Recruiters  more  likely  to  use  1-29  Navy-funded  text  message,  but  most 
feel  the  messages  are  useful  in  doing  their  job  as  a  recruiter. 


Average  Weekly  Number  of  Messages 


Usefulness  in  Doing  J  ob* 


g 

OJ  60 


1516 


El  17  16 


75 

23 

mm  rri 


Don't  1-9  msgs  10-29  30-50  50+ 

use  text  msgs  msgs  msgs 


Very  useful  Somewhat  Not  at  all 
useful  useful 


|  U  Navy- funded  B  Personal  | 

Only  includes  those  who  indicated  they  used  Navy-funded  text  messaging. 
Follow-up  Survey,  Ql-3 


On  the  follow-up  survey,  recruiters  were  asked  about  the  average  number  of 
messages  they  send  on  both  their  personal  and  Navy-funded  cell  phones;  results 
of  these  two  questions  were  used  to  determine  categorization  of  each  recruiter  by 
type  of  texter.  About  15  percent  of  recruiters  do  not  use  text  messaging  on  any  cell 
phone.  Texting  is  more  common  on  personal  phones;  over  half  of  recruiters  send 
between  1  and  29  messages  on  their  Navy  cell  phones  while  almost  half  of 
recruiters  send  30  or  more  messages  (including  31  percent  who  send  50  or  more 
messages)  weekly  on  their  personal  cell  phones. 

Respondents  who  used  Navy-funded  text  messaging  were  asked  how  useful  it 
is  in  doing  their  Navy  job.  Three-quarters  said  it  was  very  useful  while  most  of  the 
remaining  quarter  said  it  was  somewhat  useful;  only  2  percent  indicated  that  text 
messaging  was  not  at  all  useful  for  their  Navy  job. 
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Again,  there  was  little  difference  between  men  and  women  in  their  use  of  the 
Navy  cell  phone  while  there  were  larger  differences  between  officer  and  enlisted 
recruiters;  over  half  of  officer  recruiters  do  not  use  text  messaging  on  their  Navy- 
funded  cell  phones,  compared  to  23  percent  of  CPOs,  12  percent  of  POis,  and  9 
percent  of  P02s. 
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Usefulness  of  Navy- funded  Text 
Messages 


Heavy  texters  on  Navy  cell 
phones  most  likely  to  find  Navy- 
funded  messages  useful. 


Type  of  Texter 


Gender 


| a  Light  user  B  Heavy  personal  B  Heavy  Navy  □  Heavy  user  | 

Paygrade  Group 


Very  useful  Somewhat  useful  Not  at  all  useful 


Male  Female  | 


Not  at  all  useful 


ja  PQ2  a  poi  □  cpo  □  offT] 

*  Only  includes  those  who  indicated  they  used  Navy-funded  text  messaging. 

Follow-up  Survey,  Q2 


Among  the  85  percent  who  use  their  Navy  cell  phone,  there  is  a  slight 
difference  between  men  and  women  in  their  perceptions  of  usefulness  of  text 
messaging  in  doing  their  job,  with  more  women  feeling  it  is  very  useful  than  men. 
Almost  all  of  those  who  are  heavy  Navy  texters  feel  text  messaging  is  very  useful. 
As  with  prior  results,  findings  for  paygrades  show  that  officers  are  less  likely  than 
enlisted  to  feel  texting  on  their  Navy  cell  phone  is  very  useful  to  their  job, 
although  they  do  still  feel  it  is  somewhat  useful. 
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Mil 


Use  of  Personal  Cell  Phone 


Females  more  likely  to  send  many 
text  messages  on  personal  cell 
phones. 


Gender 


W  80 

cu 


21 

17  20 


1 


m  n 


Don't  use  1-9  msgs  10-29  msgs  30-50  msgs  50+  m 

|  U  Male  U  Female] 


Follow-up  Survey,  Q3 


Type  of  Texter 


1-9  msgs  10-29  msgs  30-50  msgs  50+  m 


□  P02  B  POl  □  CPO  □  Off. 


There  were  larger  differences  between  men  and  women  on  weekly  use  of  their 
personal  cell  phone  than  on  use  of  their  Navy  cell  phone.  Women  are  more  likely 
than  men  to  send  50  or  more  messages  on  their  personal  cell  phones  while  men 
are  more  likely  to  either  not  use  personal  texting  or  to  only  send  a  few  (9  or  less) 
messages.  Second  class  petty  officers  are  the  most  likely  to  send  50  or  more 
messages. 
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■Sc. 


NRC  Text  Messages  for  Policy 
Changes 


Most  believe  HQ  should  send  text  messages  for  important  policy  changes. 


100 


Should  NRC  send  text  messages  to  recruiters  to  inform  them  of 
key  changes  in  policy? 


23 

6 

Yes 


Follow-up  Survey,  Q10 


No 


Don't  know 


The  follow-up  survey  asked  respondent  opinions  on  whether  Navy  Recruiting 
Command  should  send  text  messages  directly  to  recruiters  to  inform  them  of  key 
policy  changes.  Almost  two-thirds  indicated  that  they  should.  Some  write-in 
responses  suggested  that  it  should  be  only  for  key  policy  changes  while  others 
indicated  that  it  should  be  used  as  a  supplement  to  existing  notification  methods. 
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NRC  Text  Messages  for  Policy 
Changes 


Enlisted  more  likely  to  believe 
NRC  should  send  policy- related 
text  messages  than  officers. 


Type  of  Texter 


Gender 


I 


23  26 

■ 

6  7 

Male  Female  | 


Follow-up  Survey,  Q10 


|  ■  Non-user  B  Light  user  B  Heavy  personal  P  Heavy  Navy  □  Heavy  user  | 

Paygrade  Group 


Again,  the  difference  between  men  and  women  is  small  while  the  difference 
between  officers  and  enlisted  is  much  larger.  Almost  70  percent  or  more  of  those 
who  text  at  all,  regardless  of  how  much  they  text  or  which  cell  phone  they  use  to 
text,  indicated  that  headquarters  should  send  text  messages  with  policy  changes. 
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Other  Cell  Phone-Related 
Suggestions 


Over  600  respondents  provided  suggestions  to  increase 
cell  phone  usefulness  to  recruiters 

The  large  majority  of  comments  (73%)  focused  on 
updating  the  phones,  particularly  providing: 

Bluetooth/hands-free  capabilities,  since  many  states 
require  it  when  driving 

Blackberries  or  PDA,  to  better  enable  scheduling  and 
e-mail 

QWERTY  keyboard,  for  easier  texting 

Camera,  for  photographing  tattoos 

-  GPS/mapping  functions,  for  turn-by-turn  directions  to 
recruit  homes 

Follow-up  Survey,  Qll 


The  follow-up  survey  included  an  open-ended  question  for  respondents  to 
write  in  any  other  cell  phone- related  suggestions  they  may  have  to  increase  their 
usefulness  to  recruiters.  Over  600  (46%)  provided  some  comment,  with  the 
majority  talking  about  key  components/upgrades  to  phones.  The  most  common 
upgrades  were  Bluetooth  or  other  hands-free  capabilities,  Blackberry  or  PDA 
capabilities,  full  keyboard,  camera,  and  GPS  or  mapping  functions. 
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Summary  of  NRC-specific  Findings 


Over  half  of  recruiters  text  potential  applicants  at  least 
weekly 


87%  use  text  messages  to  keep  in  touch  with  DEPers 


Only  15%  do  not  use  text  messaging  on  their  Navy  phones 

75%  of  those  who  use  indicate  it  is  "very  useful"  for  their 
job 


71%  believe  NRC  should  send  recruiters  messages  about  key 
policy  changes 


In  terms  of  findings  specific  to  the  Navy  Recruiting  Command,  four  key  points 
stand  out.  Clearly,  text  messaging  is  being  used  by  recruiters,  with  over  half 
texting  potential  applicants  at  least  weekly  and  87  percent  using  text  messages  to 
keep  in  contact  with  DEPers;  only  15  percent  do  not  use  text  messaging  on  their 
Navy  phones.  Of  those  who  do,  75  percent  find  it  to  be  “very  useful”  and  only  2 
percent  indicate  it  is  not  useful  to  them.  The  large  majority  of  respondents 
believe  that  headquarters  should  send  messages  directly  to  recruiters  about  key 
policy  changes. 


32 


■Sc,  I 


NRC-Specific  Recommendations  (i 

of  2) 


Continue  allowing  text  messaging  on  Navy  phones 

-  Evaluate  possibility  of  increasing  limit,  especially  if 
institutionalizing  use  of  text  messaging  to  market  to 
Millennial 

Consider  upgrading  phones,  using  those  better  designed  for 
text  messaging  (e.g.,  those  with  full  keyboard)  if  continuing 
to  allow  text  messaging 

Create  training  module  and  accompanying 
guidebook/pamphlet  on  appropriate  use  of  text  messaging 
for  recruiting 

Provide  suggestions  of  how  to  use  for  marketing  (e.g., 
"The  Navy  ROTC  college  scholarship  web  site  is  open  for 
applications.  Go  to  www.nrotc.navv.com”) 


Considering  the  key  findings,  the  primary  recommendation  to  NRC  is  to 
continue  allowing  text  messaging  on  Navy  phones  and  supporting  their  use  for 
recruiting  purposes.  Prior  to  the  survey  results  being  briefed,  the  limit  was  lifted 
and  most  recruiters  now  receive  an  unlimited  allowance  of  messages. 

Another  consideration  would  be  to  upgrade  recruiter  cell  phones,  perhaps 
selecting  phones  better  designed  for  text  messaging.  Navy  Recruiting  Command 
has  been  having  discussions  with  other  Navy  stakeholders  about  a  hand-held  tool 
(such  as  a  cell  phone)  that  could  be  better  used  by  recruiters  for  a  number  of 
purposes. 

In  embracing  text  messaging  for  recruiting  purposes,  a  training  module  and 
accompanying  guidebook/pamphlet  should  be  created  to  provide  recruiters  with 
suggestions  on  how  to  best  use  text  messages  for  marketing.  The  example 
provided  above  regarding  the  Navy  ROTC  college  scholarship  web  site  was 
actually  sent  from  a  recruiter  in  response  to  one  of  the  cell  phone  questions  as  an 
example  of  how  he/she  uses  it  to  market  Navy  programs. 
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NRC-Specific  Recommendations  (2 

of  2) 

> 

•  Utilize  text  messaging  from  NRC  for  KEY  policy  updates 

-  Cpnsider  asking  cell  phone  contractor  to,  preprogram  NRC 
phone  number  so  recruiters  automatically  know  message 
is  from  NRC  HQ  y  a 

•  Consider  utilizing  text  messaging  for  additional  data 
collection 

Recruiter  Lightning  Poll  Panel  -  representative  group  of 
recruiters  wno  could  be  pulsed  several  times  a  year  on  key 
issues 

Extend  Lightning  Poll  to  DEPers 


Mi C, 

|  * 


Provide  feedback.letter  to  respondents  about  changes  based 
upon  survey  results 


Since  the  majority  of  respondents  indicated  that  text  messaging  should  be 
used  for  key  policy  updates,  NRC  should  consider  investing  more  in  doing  this. 

To  ensure  that  recruiters  know  that  the  message  is  from  headquarters,  the  cell 
phone  contractor  should  preprogram  the  NRC  number  into  each  phone  before 
issuing  it  to  the  recruiter. 

If  making  the  investment  in  text  messaging  at  NRC,  there  are  additional  data 
collection  options  that  might  be  considered.  For  example,  a  recruiter  panel  might 
be  possible,  allowing  headquarters  to  periodically  question  recruiters  on  key 
issues  throughout  the  year  while  only  having  to  train  them  once  on  how  to  do  a 
text  message  survey.  Additionally,  surveys  of  DEPers  might  be  possible  through 
this  technique. 

A  reason  respondents  do  not  complete  surveys  is  because  of  a  perceived  lack 
of  impact  of  the  results  on  policies  and  programs.  A  suggestion  to  increase 
response  rates  is  to  provide  survey  feedback  of  the  compiled  results  and  any 
planned  changes  in  policies  and  programs  (Uriell  &  Schultz,  2008).  One 
recommendation  for  the  Lightning  Poll  would  be  to  provide  a  feedback  letter  to 
respondents  to  let  them  know  about  any  changes  being  considered  or  made  based 
upon  the  survey  results. 
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In  addition  to  the  findings  specific  to  the  Navy  Recruiting  Command,  the 
current  survey  effort  provides  results  relevant  to  Lightning  Polls  and  other  text 
message-based  surveys. 
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Specific  Issues  to  Research  about 
Data  Collection  by  Lightning  Poll 

•  Are  there  age  differences  in  access  to  and  use  of  cell  phones?  0 

•  How  do  results  compare  to  larger  Navy-wide  surveys? 

•  Can  SMS  be  used  aboard  ship  or  overseas,  or  is  it  just  for  shore 
CONUS  units? 

•  Are  there  age/gender  differences  in  responses?  Completion  rates 
(especially  for  juniors,  who  are  least  likely  to  complete  traditional 
surveys)?  0 

•  How  long  does  it  take  for  first  answer?  To  complete  survey?  0 

•  How  should  it  be  done  (programming  issues,  across  cell  company 
issues,  etc)?  0 

•  What  are  attitudes  towards  cell  phone  survey  (length,  desire  to  do 
again,  etc.)?  0 


Because  the  Lightning  Poll  is  a  research  effort,  a  number  of  methodological 
issues  are  being  investigated  including  those  mentioned  above.  The  NRC  poll 
addressed  some  of  those  questions,  as  shown  above  by  checkmarks,  with  grey 
checks  indicating  a  partial  answer  and  black  indicating  a  complete  answer.  An 
additional  Lightning  Poll  was  conducted  to  answer  the  remaining  questions. 
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Forty-nine  percent  of  recruiters  responded  to  the  first  text  message  question; 
however,  some  of  the  initial  responses  were  “Who  is  this.”  Only  35  percent 
provided  interpretable  answers  to  both  questions,  which  is  in  line  with  response 
rates  of  other  surveys  (e.g.,  Whittam,  2008;  Newell,  Whittam,  &  Uriell,  2008). 

On  the  follow-up  survey,  respondents  were  asked  if  they  had  completed  the  text 
message  survey.  Sixty-three  percent  indicated  that  they  had,  inconsistent  with 
both  the  49  percent  rate  of  those  who  responded  in  some  way  to  the  first  question 
and  the  35  percent  rate  of  those  who  completed  the  entire  survey.  This 
inconsistency  is  similar  to  findings  from  other  non-response  follow-up  surveys, 
where  more  people  reported  they  had  responded  to  the  initial  survey  than 
actually  had  (Uriell  &  Schultz,  2008). 
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Percent  Who  Completed  Lightning 
Poll  -  Actual 


Males  and  those  who  are  heavy 
texters  on  both  personal  and 
Navy  cell  phones  were  most  likely 
to  complete  Lightning  Poll. 


Gender 


Type  of  Texter 


_  60  58 

42  46  47  :  ■ :  ■ 

MU 


|  ■  Non-user  E  Light  user  B  Heavy  personal  □  Heavy  Navy  □  Heavy  u 

Paygrade  Group 


[■Male  ■  Female |  □  PQ2  ■  POl  □  CPO  tl Off^ 

*  Actual  numbers  based  upon  those  who  provided  two  answers.  "Don't  know"  option  not  possible. 


Actual  completion  rates  for  the  Lightning  Poll  were  slightly  different  between 
men  and  women,  with  men  slightly  more  likely  to  complete  both  questions  than 
women.  Not  surprisingly,  those  who  are  most  proficient  with  text  messaging, 
both  on  personal  and  Navy  cell  phones,  were  more  likely  to  complete  the 
Lightning  Poll  than  those  who  indicated  that  they  do  not  use  text  messaging. 
Contrary  to  some  of  the  findings  discussed  earlier  that  show  officers  as  less  likely 
to  use  text  messaging,  completion  rates  across  paygrades  did  not  substantially 
differ. 
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Survey  Ease  and  Likelihood  to  Do 
Other  Surveys 


Almost  all  of  those  who  completed  the  Lightning  Poll  thought  it  was  easy 
and  would  be  likely  to  do  other  short  SMS  surveys. 


Was  the  survey  easy  to  do? 


Would  you  be  likely  to  complete 
other  short  SMS  surveys? 


Yes  No  No  opinion  Likely 

*  Results  only  for  those  who  provided  two  answers  to  the  Lightning  Poll. 
Follow-up  Survey,  Q5  &  Q8 


Unlikely 


Several  questions  on  the  follow-up  survey  asked  about  opinions  of  the 
Lightning  Poll  process.  Two  questions  related  to  the  ease  of  doing  the  survey  and 
likelihood  of  doing  other  text  message  surveys.  Results  are  presented  only  for 
those  who  actually  completed  the  Lightning  Poll  by  answering  both  questions. 
Ninety-five  percent  indicated  that  they  felt  the  survey  was  easy  to  do;  only  l 
percent  indicated  that  it  was  not.  Over  three-fourths  indicated  that  they  would  be 
likely  to  do  another  text  message  survey,  while  only  6  percent  said  they  were 
unlikely  to  do  another. 
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Lightning  Poll  Compared  to  Other 
Navy  Surveys 


Lightning  Polls  more  convenient  and  as  secure  as  other  Navy  surveys. 


Was  the  survey  more  convenient  than 
other  Navy  surveys  you  have  done? 


Did  you  feel  your  responses  were 
more  or  less  secure  than  other 
Navy  surveys? 


r 

21 

t 

■ 

3 

No  difference 


More  secure  No  difference  Less  secure 


Results  only  for  those  who  provided  two  answers  to  the  Lightning  Poll  and  had  participated  in  other  Navy  surveys. 
Follow-up  Survey,  Q6  &  Q7 


Because  Lightning  Polls  are  being  researched  as  a  possible  tool  in  the  Data 
Collection  Alternatives  toolbox,  two  questions  on  the  follow-up  survey  asked  how 
the  Lightning  Poll  compared  to  other  Navy  surveys,  in  terms  of  both  convenience 
and  the  perceived  security  of  responses.  Again,  responses  are  only  displayed  for 
those  who  had  replied  to  the  Lightning  Poll  while  narrowing  further  to  only  those 
who  reported  that  they  had  completed  other  Navy  surveys.  Three-quarters  felt 
that  the  survey  was  more  convenient  than  other  Navy  surveys,  while  21  percent 
felt  it  was  no  different.  Responses  were  perceived  to  be  as  secure  as  other  Navy 
surveys;  about  the  same  percentage  indicated  it  was  more  secure  as  indicated  it 
was  less  secure. 
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Response  Times 


Response  times  should  be  interpreted  cautiously 

Seeming  overload  due  to  number  of  initial  messages  led  to  multiple 
transmissions  of  same  message;  time  of  first  answer  used  in  analyses 

Software  saved  message  creation  time  instead  of  message  sending 
time,  and  some  messages  were  not  sent  until  3  days  after  they  were 
created 


Response  Time  for  All  Respondents  (in  minutes) 

Group 

N 

Average 

Shortest 

Longest 

Any  response  to 
first  question 

2,259 

2,940  (~  2d) 

2 

12,363 

Answer  to  first 
question 

2,028 

2,978  (~  2d) 

2 

12,363 

Answer  to  second 
question 

1,639 

1,569  (~  Id) 

0* 

8,259 

Survey  Completion 
(2  answers) 

1,602 

4,587  (~  3d) 

53 

12,378 

:  Half  responded  within  one  minute,  with  additional  40%  responding  within  10  minutes  of  receiving  second  question. 


One  of  the  features  of  a  Lightning  Poll  that  makes  it  attractive  is  the  potential 
for  rapid  turnaround  of  responses.  Analyses  were  done  of  response  times,  but 
results  should  be  interpreted  cautiously  because:  (1)  the  system  was  overloaded 
with  4,620  initial  messages,  which  caused  a  delay  in  sending  the  second  question 
for  some;  and  (2)  the  software  downloaded  the  time  the  message  was  added  to 
the  queue,  as  opposed  to  the  time  it  was  actually  sent  from  the  computer,  which 
may  lead  to  long  response  times  for  the  first  question. 

The  table  shows  response  times  in  minutes  for  all  respondents.  The  response 
time  to  answer  anything  to  the  first  question  ranged  from  2  minutes  to  12,363 
minutes,  with  an  average  of  about  2  days.  The  response  time  to  answer  the 
second  question  was  much  shorter,  with  over  half  of  respondents  answering 
within  one  minute  and  a  total  of  90  percent  answering  within  10  minutes  of  the 
second  question  being  queued. 

Time  for  survey  completion  (defined  as  two  useable  answers)  ranged  from  53 
minutes  to  12,378  minutes,  with  the  average  being  about  3  days. 
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Respondents  were  given  no  guidance  in  the  notification  letter  of  how  to 
complete  the  survey.  The  largest  percentage  of  respondents  provided  a  single 
letter  answer,  more  so  for  the  second  question  (59%)  than  the  first  (47%).  The 
next  most  common  (12%  for  both  questions  1  and  2)  answer  type  was  providing  a 
letter  either  before  or  after  the  entire  question,  with  non-texters  more  likely  to  do 
this  (24%)  than  those  who  were  heavy  texters  (4%).  Another  8  percent  provided 
the  answer  letter  as  well  as  a  written  answer.  Six  percent  of  those  who  responded 
to  the  first  question  (3%  of  those  responding  to  the  second)  opted  to  provide  a 
modified  type  of  answer,  similar  to  having  a  write-in  option. 
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Suggestions  to  Improve  the  SMS 
Survey  Process 

•  254  respondents  provided  comments  relating  to  the 
SMS  survey  process 


The  largest  group  of  responses  (35%)  indicated  that  the 
process  worked  great  as  it  was 


Some  (18%)  indicated  that  advance  notification  was  key 


12%  indicated  that  they  had  technical  problems,  such 
as  getting  multiple  of  the  same  message 


Follow-up  Survey,  Q9 


Respondents  were  given  the  opportunity  to  provide  input  about  how  to 
improve  the  SMS  survey  process.  The  largest  group  of  responses  indicated  that 
recruiters  thought  that  the  process  worked  well  as  it  was.  Another  18  percent 
indicated  that  advance  notification  was  important,  an  issue  made  clear  by 
recruiters  whose  first  response  was  “Who  is  this?”  An  additional  12  percent  talked 
about  technical  problems  they  experienced  such  as  getting  multiple  copies  of  the 
same  message. 
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Lightning  Poll-Specific  Summary 


Kil 


Text  survey  considered  easy  to  do  by  95%  of  follow-up 
respondents  who  did  the  Lightning  Poll 

Over  3/4  indicated  they  would  be  likely  to  complete 
other  short  SMS  surveys 


Lightning  Poll  considered  more  convenient  than  other 
Navy  surveys 


75%  feel  responses  on  SMS  surveys  are  as  secure  as 
other  Navy  surveys 


In  summary,  results  from  the  Navy  Recruiting  Command  Lightning  Poll  show 
that  most  feel  that  the  Lightning  Poll  was  easy  to  do  and  most  would  be  likely  to 
complete  other  short  SMS  surveys.  Lightning  Polls  are  considered  to  be  more 
convenient,  while  still  as  secure  as  other  Navy  surveys. 

These  findings  expand  the  paradigm  of  Navy  personnel  surveys  to  an 
administration  medium  that  the  military  had  never  before  used  for  this  purpose  . 
Based  on  this  initial  study,  it  appears  that  Lightning  Polls  are  a  viable  data 
collection  alternative  for  use  with  select  Navy  populations.  Future  research 
efforts  will  be  needed  to  further  investigate  the  reliability  and  validity  of 
Lightning  Polls  as  an  assessment  tool  in  other  operational  Navy  settings. 
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Recommendations  for  Future 
Lightning  Polls  (i  of  2) 

> 

•  Conduct  additional  Lightning  Polls  to  answer  additional 
research  questions 

Difference  in  demographics  of  respondents 
Comparison  to  other  survey  methods 

•  Ensure  adequate  notification 

-  Consider  using  panel  format,  where  respondents  are 
selected  and  notified  once  but  are  requested  to  respond  to 
successive  surveys  as  needed 


n.  •  Consider  explanation  that  system  is  automated 

Some  respondents  may  not  have  received  initial 
notification  in  time  so  thought  that  the  messages  they 
received  were  from  potential  applicants 


Results  specific  to  Lightning  Polls  suggest  some  areas  for  improvement.  First, 
additional  research  needs  to  be  completed  to  determine  if  there  are  other 
demographic  differences  (e.g.,  by  age,  by  marital  status)  and  if  results  are 
comparable  to  other  survey  methods  currently  in  place  in  the  Navy.  Second, 
additional  steps  are  necessary  to  ensure  adequate  notification,  with  one  option 
being  to  utilize  a  panel  format  that  allows  for  successive  surveys  of  the  same 
person  while  only  needing  to  explain  the  details  the  first  time.  Third,  the 
notification  should  perhaps  indicate  that  the  system  is  automated  to  minimize 
the  number  of  answer  responses  that  are  modified  (i.e.,  write-in  answers). 


I- 

in 

a: 
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Recommendations  for  Future 
Lightning  Polls  (2  of  2) 

•  Provide  guidelines  for  completion  ("a"  vs  "agree”  vs  long 
clarifying  responses) 

•  Send  initial  messages  in  smaller  sets  (to  avoid  overloading 
system) 

1- 

•  Vary  answers  between  questions  to  avoid  confusion  about 
which  answer  goes  to  which  question  (i.e.,  a/b/c  for  question 

1  and  x/y/z  for  question  2) 

in 

a: 

•  Contact  cell  phone  companies  to  allow  for  caller  ID  of 

Lightning  Poll  number 

1 

Additionally,  guidelines  for  completing  the  survey  would  ease  analyses. 
Because  the  system  sends  each  message  separately  and  messages  are  processed 
First-In-First-Out,  smaller  groups  of  initial  messages  would  potentially  have 
avoided  overloading  the  system  and  allowed  faster  response  times.  Problems  with 
notification  led  to  mistiming  of  responses  and  thereby  difficulties  in  determining 
which  answer  was  for  what  question,  an  issue  that  could  have  been  avoided  if 
answer  letters  varied  from  question  to  question.  Notification  problems  also  led  to 
confusion  regarding  who  was  sending  the  questions;  allowing  caller  ID  of  the 
survey  phone  number  might  have  alleviated  some  of  the  confusion. 
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Appendix  A: 
Backup  Slides 


A-o 


A-i 


Percent  Who  Completed  Lightning 
Poll  -  Self-Report 


Males  and  heavy  texters  are  more 
likely  to  believe  they  completed 
the  Lightning  Poll. 
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Follow-up  Survey,  Q4 


Survey  Ease 
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No  difference  between  subgroup 
on  perceived  ease  of  Lightning 
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Likelihood  to  Do  Other  Surveys 


Males,  heavy  users,  and  junior 
personnel  likely  to  do  other 
Lightning  Polls. 
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Follow-up  Survey,  Q8 


Convenience  of  Lightning  Poll 
Compared  to  Other  Navy  Surveys 


Females  and  officers  more  likely 


to  say  that  there  is  no  difference 
in  convenience  between  Lightning 
Poll  and  other  Navy  surveys. 
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*  Results  only  for  those  who  provided  two  answers  to  the  Lightning  Poll  and  had  participated  in  other  Navy  surveys. 
Caution  due  to  low  numbers  in  subgroups. 

Follow-up  Survey,  Q6 
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Security  of  Lightning  Poll 
Compared  to  Other  Navy  Surveys 


Heavy  texters  overall  and  heavy 
personal- phone  texters  most 
likely  to  indicate  no  difference  in 
security  of  Lightning  Polls  as 
compared  to  other  surveys. 
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*  Results  only  for  those  who  provided  two  answers  to  the  Lightning  Poll  and  had  participated  in  other  Navy  surveys. 
Caution  due  to  low  numbers  in  subgroups. 

Follow-up  Survey,  Q7 


Other  Observations 
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•  Advance  notice  key 

Some  responded  to  first  question  with  "Who  is  this 
and  are  you  interested  in  the  Navy?" 
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Some  left  cell  phones  on  overnight  so  received  early 
wakeup  from  arriving  text  messages 

Clearly  an  issue  for  Hawaii  personnel 


Response  Time  Analyses 


Software  saved  message  creation  time  instead  of  message  sending 
time 

System  seemingly  became  overloaded,  so  some  messages  were 
not  sent  until  3  days  after  they  were  created 


Seeming  overload  also  led  to  multiple  transmissions  of  same 
message 

Time  of  initial  transmission  of  question  used  in  analyses 


Response  times  to  initial  question  and  overall  survey  time  should 
be  considered  with  caution  due  to  overload  situation 
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Time  to  Respond  to  Second 
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Time  to  Respond  to  Both 
Questions  (Complete  survey) 
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Appendix  B: 

Notification  Letter  and  Privacy  Act 


B-o 


CNRC  LETTERHEAD 


«name» 

<<addressi>> 

<<address2>> 

<<address3>> 

<<address4>> 

Dear  «name», 

As  a  recruiter,  you  have  the  unique  opportunity  to  participate  in  the  first- 
ever  Navy  survey  conducted  using  text  messaging.  The  survey  will  consist  of  only 
two  questions  about  your  use  of  text  messaging.  The  questions  will  be  sent  one  at 
a  time  from  (901)  451-1522.  To  respond,  simply  reply  to  the  text  message  with 
the  letter  of  your  answer  choice. 

Shortly  after  the  text  messaging  survey  is  conducted,  you  will  receive  a 
follow-up  survey  asking  your  opinion  of  the  usefulness  of  text  messaging  surveys. 
This  survey  should  take  about  10  minutes  of  your  time. 

While  participation  is  voluntary,  your  feedback  is  vital  in  providing 
accurate  data  about  text  messaging  by  recruiters.  Be  assured  that  the 
information  you  give  will  not  be  attributed  to  you  personally,  nor  will  it  have  a 
direct  impact  on  your  Navy  career.  The  Navy  Personnel  Research,  Studies,  and 
Technology  (NPRST)  Department  will  only  provide  me  with  numerical  results;  no 
personal  identifiers  will  be  included  to  maintain  your  anonymity.  Please  see  the 
enclosed  Privacy  Notice  for  more  details. 

Thank  you  in  advance  for  all  you  do  for  the  Fleet.  If  you  have  any 
questions,  please  contact  the  Project  Director,  Zannette  Uriell,  at  DSN  882-4641, 
(901)  874-4641,  zannette.uriell@navy.mil. 


Sincerely, 


R.  R.  BRAUN 
Rear  Admiral,  U.S.  Navy 
Deputy  Commander 
Navy  Recruiting  Command 
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PRIVACY  NOTICE 


Authority  to  request  this  information  is  granted  under  Title  5,  U.S.  Code  301,  and 
Department  of  the  Navy  Regulations.  License  to  administer  this  survey  is 
granted  under  OPNAV  Report  Control  Symbol  1100-2,  which  expires  on  31  March 
2010. 

PURPOSE:  The  purpose  of  this  questionnaire  is  to  collect  data  to  evaluate 
existing  and  proposed  Navy  personnel  policies,  procedures,  and  programs. 

ROUTINE  USES:  The  information  provided  in  the  short  message  service  (SMS) 
questionnaire  will  be  analyzed  by  the  Navy  Personnel  Research,  Studies,  and 
Technology  Department.  The  data  files  will  be  maintained  by  the  Navy  Personnel 
Research,  Studies,  and  Technology  Department  where  they  may  be  used  for 
determining  changing  trends  in  the  Navy. 

ANONYMITY:  All  responses  will  be  held  in  confidence  by  the  Navy  Personnel 
Research,  Studies,  and  Technology  Department.  Information  you  provide  will  be 
considered  only  when  statistically  summarized  with  the  responses  of  others,  and 
will  not  be  attributable  to  any  single  individual. 

PARTICIPATION:  Completion  of  this  questionnaire  is  entirely  voluntary. 

Failure  to  respond  to  any  of  the  questions  will  NOT  result  in  any  penalties  except 
possible  lack  of  representation  of  your  views  in  the  final  results  and  outcomes. 
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Appendix  C: 

SMS  Follow-on  Survey 


C-i 


1.  How  often  do  you  text 
one  of  your  potential 
applicants? 
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SMS  Follow-up  Survey 


/  PRIVACY  ACT  STATEMENT 

Authority  to  request  this  information  is  granted  under  Title  5,  U.S.C.  301 ,  and 
Department  of  the  Navy  Regulations.  License  to  administer  this  survey  is  granted 
per  OPNAVINST  5300. 8B  under  OPNAV  Report  Control  Symbol  1100-2,  which 
expires  on  31  March  2010. 

PURPOSE:  The  purpose  of  this  questionnaire  is  to  collect  information  to  evaluate 
existing  and  proposed  Navy  personnel  policies,  procedures,  and  programs. 

ROUTINE  USES:  The  information  provided  in  the  Short  Message  Service  (SMS) 
questionnaire  will  be  analyzed  by  the  Navy  Personnel  Research,  Studies,  and 
Technology  Department.  The  data  files  will  be  maintained  by  the  Navy  Personnel 
Research,  Studies,  and  Technology  Department  where  they  may  be  used  to 
determing  changing  trends  in  the  Navy. 


a)  Daily 

b)  Weekly 

c)  Monthly 

d)  Rarely 

e)  Never 
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ANONYMITY:  All  responses  will  be  held  in  confidence  by  the  Navy  Personnel  Research,  Studies,  and 
Technology  Department.  Information  you  provide  will  be  statistically  summarized  with  the  responses  of 
others,  and  will  not  be  attributable  to  any  single  individual. 


PARTICIPATION:  Completion  of  this  questionnaire  is  entirely  voluntary.  Failure  to  respond  to  any  of 
the  questions  will  NOT  result  in  any  penalties  except  for  lack  of  representation  of  your  views  in  the  final 
results  and  outcomes. 
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IMPORTANT  MARKING  INSTRUCTIONS 


•  Use  black  or  blue  pen  or  a  No.  2  pencil 

•  Make  dark  marks  that  stay  within  the  box 

•  Make  no  stray  marks 

•  Do  NOT  fold,  tear,  or  mutilate  this  form. 


CORRECT:  S^B 


If  you  have  any  questions,  please  contact  the  Project  Director,  Zannette  Uriell,  at  DSN 
882-4641 ,  (901 )  874-4641 ,  zannette.uriell@navv.mil. 


Department  of  the  Navy 
Bureau  of  Naval  Personnel 

Navy  Personnel  Research,  Studies  and  Technology  (NPRST/PERS-1) 
5720  Integrity  Drive 
Millington,  TN  38055-1000 


N  P  RS 


research 
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PLEASE  DO  NOT  WRITE  IN  THIS  AREA 

- C=4 - 1 


[SERIAL] 


1 .  On  average,  how  many  text  messages  do  you  send  on 
your  NAVY  cell  phone  per  week? 

My  phone  has  text-message  capability  but  I  don't  use  it 
1-9  messages 
10-29  messages 
30-50  messages 
More  than  50  messages. 


□ 

□ 

□ 

□ 

□ 


2.  How  useful  are  those  Navy-funded  text  messages  in 
doing  your  job? 

□  N/A;  I  don't  use  text  messaging 

□  Very  useful 

□  Somewhat  useful 

□  Not  at  all  useful 

3.  On  average,  how  many  text  messages  do  you  send  on 
your  PERSONAL  cell  phone  per  week? 

□  I  don't  have  a  personal  cell  phone 

□  My  phone  has  text-messaging  capability  but  I  don't  use  it 

□  1-9  messages 

□  10-29  messages 

□  30-50  messages 

□  More  than  50  messages 

4.  Did  you  complete  the  recent  CNRC  SMS  Survey? 


□ 

□ 

□ 


Yes 

No  - 

Don't  know 


Skip  to  question  10 


5.  Was  the  survey  easy  to  do? 


□ 

□ 

□ 


Yes 

No 

No  opinion 


6.  Was  the  survey  more  convenient  than  other  Navy  surveys 
you  have  done? 

□  N/A,  have  never  participated  in  a  Navy  survey 

□  Yes,  more  convenient 

□  No  difference  from  other  Navy  surveys 

□  No,  less  convenient 

7.  Did  you  feel  your  responses  were  more  or  less  secure 
than  other  Navy  surveys? 

□  N/A,  have  never  participated  in  a  Navy  survey 

□  More  secure 

□  No  difference  in  security 

□  Less  secure 


8.  Would  you  be  likely  to  complete  other  short  (5  questions 
or  less)  SMS  surveys? 

□  Yes,  likely  to  complete  other  SMS  surveys 

□  Neither  likely  nor  unlikely  to  complete  other  SMS 
surveys 

□  No,  unlikely  to  complete  other  SMS  survey 

9.  How  do  you  think  the  SMS  survey  process  can  be 
improved? 
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10.  Should  CNRC  send  text  messages  to  recruiters  to  inform 
them  of  key  changes  in  policy? 


40] 


□ 

Yes 

□ 

No 

□ 

Don't  know 

[37] 


[® 


11.  Do  you  have  any  cell  phone-related  suggestions  to 
increase  their  usefulness  to  you  as  a  recruiter? 


THANK  YOU! 


Please  complete  the  survey  as  soon  as  possible,  and 
return  in  the  enclosed  envelope  to: 

Survey  Operations  Center  (SOC) 

Navy  Personnel  Research,  Studies,  and  Technology 
(PERS-14) 

5720  Integrity  Drive 
Millington,  TN  38055-1400 

Socform:  EMC0801 
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Distribution 


AIR  UNIVERSITY  LIBRARY 

ARMY  RESEARCH  INSTITUTE  LIBRARY 

ARMY  WAR  COLLEGE  LIBRARY 

CENTER  FOR  NAVAL  ANALYSES  LIBRARY 

HUMAN  RESOURCES  DIRECTORATE  TECHNICAL  LIBRARY 

JOINT  FORCES  STAFF  COLLEGE  LIBRARY 

MARINE  CORPS  UNIVERSITY  LIBRARIES 

NATIONAL  DEFENSE  UNIVERSITY  LIBRARY 

NAVAL  HEALTH  RESEARCH  CENTER 

NAVAL  POSTGRADUATE  SCHOOL  DUDLEY  KNOX  LIBRARY 

NAVAL  RESEARCH  LABORATORY  RUTH  HOOKER  RESEARCH  LIBRARY 

NAVAL  WAR  COLLEGE  LIBRARY 

NAVY  PERSONNEL  RESEARCH,  STUDIES,  AND  TECHNOLOGY  SPISHOCK 
LIBRARY  (3) 

NAVY  RECRUITING  COMMAND  (N5) 

PENTAGON  LIBRARY 

USAF  ACADEMY  LIBRARY 

US  COAST  GUARD  ACADEMY  LIBRARY 

US  MERCHANT  MARINE  ACADEMY  BLAND  LIBRARY 

US  MILITARY  ACADEMY  AT  WEST  POINT  LIBRARY 

US  NAVAL  ACADEMY  NIMITZ  LIBRARY 


